Spatio-Temporal Light Field Cameras

ABSTRACT

Spatio-temporal light field cameras that can be used to capture the light field within its spatio temporally extended angular extent. Such cameras can be used to record 3D images, 2D images that can be computationally focused, or wide angle panoramic 2D images with relatively high spatial and directional resolutions. The light field cameras can be also be used as 2D/3D switchable cameras with extended angular extent. The spatio-temporal aspects of the novel light field cameras allow them to capture and digitally record the intensity and color from multiple directional views within a wide angle. The inherent volumetric compactness of the light field cameras make it possible to embed in small mobile devices to capture either 3D images or computationally focusable 2D images. The inherent versatility of these light field cameras makes them suitable for multiple perspective light field capture for 3D movies and video recording applications.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a divisional of U.S. patent application Ser. No.13/659,776 filed Oct. 24, 2012 which claims the benefit of U.S.Provisional Patent Application No. 61/654,688 filed Jun. 1, 2012.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates to the field of light field cameras,radiance cameras, directional light capture, 3D cameras and 2D/3Dswitchable cameras.

2. Prior Art

The advent of nano-scale semiconductors is making it possible to havesufficient computing resources within a typical mobile device, such as acell phone or a tablet PC, for example, to process high resolutionvisual information received and/or transmitted through the high datarate mobile networks such devices can typically connect to. A typicalmobile device now has a 1M pixel display and an 8M pixel camera allowingthe mobile user to view and capture high resolution visual information.With such visual communication capabilities, mobile devices are on theverge of being capable of processing 3D visual information. Although thecapture and recording of 2D visual information by ultra compact camerasthat meet the stringent volumetric constraints of the mobile devices isnow main stream, such is not the case for the capture and recording of3D visual information. Even in the case of the capture and recording of2D visual information, the stringent volumetric constraints of themobile devices is still making it difficult to have cameras withadvanced auto focusing capabilities embedded in such devices. The mainreason being the bulkiness, poor performance and excessive added cost ofincorporating advanced auto focusing features into cameras targeting themobile devices. A promising prospect for possibly overcoming some ofthese constraints is a class of cameras known as light field cameraswhich are capable of capturing information about the directionaldistribution of the light rays that enter the camera. Besides providingthe mobile user with the ability to capture 3D images, the ability tocapture the directional information of the light would also enablecomputational (digital) focusing which would allow the mobile user tocapture the entire light field without regard to focusing, then leveragethe ample processing capabilities of the mobile device tocomputationally focus on any desired aspects of the captured lightfield. In that regard a light field mobile camera would in effectleverage the abundant processing resources now becoming typical inmobile devices to get rid of the expensive and bulky auto focusing. Theproblem in realizing such a prospect, however, is that the currentstate-of-the-art light field cameras are inherently bulky in themselvesand not at all suited for being embedded in mobile devices. Beforeproceeding to describe the details of the current invention, thefollowing discussion puts into perspective the current state-of-the-artof light field cameras approaches and their salient characteristics.

Conventional cameras do not record the directional information of thelight it captures. A conventional camera captures only a two-dimensional(2D) image that represents a one to one correspondence in lightoriginating from a point in the viewing scene to a corresponding spatialposition (pixel) on its photo-detector (PD), as such spatial informationis captured but all of the directional information is lost. In contrastto conventional 2D cameras, light field cameras capture both the spatialas well as the directional information of the light. Light field camerasare able to capture both spatial and directional information of thelight because they are able to record the radiance of the light, whichdescribes both spatial and directional (angular) information, and isdefined as the radiant flux of the incident light per unit of area perunit of solid angle (measured in W·m⁻²·Sr⁻¹). A light field camera,therefore, is able to sample the four-dimensional (4D) radiance, in sodoing captures both the two dimensions of spatial and the two dimensionsof directional distributions of the light it captures. Being able torecord the radiance, a light field camera therefore captures all of thelight field information needed to post-capture focusing, reduce thenoise, or change the viewpoint; i.e., three-dimensional (3D) imagecapture.

FIG. 1A illustrates a prior art light field camera implemented using anarray of conventional cameras whereby each of the cameras records animage of the light field from a different perspective. The capturedimages may then be combined to form the captured light field. Thedrawbacks of this approach are rather obvious; in order to capture areasonable angular extent with each camera in the array, the array ofobjective lenses will span a much larger area than their photo-detectorsand will each have a rather large optical track length, thus making thewhole camera array of FIG. 1A be limited in terms of the number of viewsof the light field it can capture, and excessively bulky, thus not atall suitable for embedding in mobile devices.

FIG. 1B illustrates another prior art light field camera implementedusing the principal of integral imaging. In this light field cameraapproach, which is also known as a plenoptic camera, only one objectivelens is used and a lenslet or micro lens array is placed near the cameraphoto-detector to sample the aperture of the camera. The image capturedby the plenoptic camera would be made up of an array of sub-apertureimages of the light field each recorded by the group of pixelsunderneath each of the micro lens elements. Each of the sub-apertureimages captured by the plenoptic camera would represent a parallaxsample of the light field. Although the plenoptic camera of FIG. 1Bwould potentially provide a higher number of views of the light fieldand would also be volumetrically smaller than the camera array of FIG.1A, the increase in the number of views would be at the expense ofreduced spatial resolution. In addition, similar to the camera array,for the plenoptic camera to cover a reasonable angular extent, it mustemploy an as large as possible diameter objective lens which in turnrequires a large optical track length, thus make the plenoptic cameraalso bulky and not at all suitable for embedding in mobile devices.

FIG. 1C illustrates yet another prior art light field camera implementedby using the principal of frequency domain analysis of the light field.In this type of prior art field camera, which although is conceptuallyequivalent to the plenoptic camera of FIG. 1B, for differentiation willbe referred to as radiance camera, is implemented by placing anon-refractive two-dimensional array of pinholes, basically a mask,either in front of the objective lens or in between the main lensassembly and the photo-detector of an otherwise conventional camera. Theimage captured by such a camera is, therefore, a Fourier domainconvolution of the incoming light field with the known non-refractivelight field modulation weighting function of the mask. This cameraactually captures the 4-D light field directly in the Fourier domain,thus the values recorded by each pixel of the 2-D photo-detector of thecamera represents a coded linear combination in the Fourier domain ofall the rays entering the camera from multiple directions. The knownlinear combination superimposed by the non-refractive mask light fieldcan be decoded by software to obtain the 4-D light field. In general theperformance of this radiance camera is similar in terms the spatial anddirectional resolution it can achieve using a given photo-detector size,in terms of number of pixels, except that the radiance analysis cameramay offer increased spatial resolution per view, but the number of viewsthat can be resolved is highly dependent on the computational throughputone is willing to allocate to the post-capture processing. In otherwords, the improvement in the spatial resolution per view that may beoffered by the radiance camera would be at the expense of increasedcomputational resources. Furthermore, the mask used in the radiancecamera will cause light loss that would tend to reduce the capture imagesignal to noise ratio (SNR). In addition, similar to the camera arrayand the plenoptic camera, for the radiance camera to cover a reasonableangular extent, it must employ as large as possible diameter objectivelens which in turn requires a large optical track length, thus makingthe radiance analysis camera also bulky and not at suitable forembedding in mobile devices.

In general, prior art light field cameras illustrated in FIGS. 1A, 1Band 1C are limited in their functionality and applications because:

1. The depth of their light field is limited by the focus depth of theirobjective lens;

2. The field of view of their light field is limited by the angularextent of their objective lens;

3. Their objective lens and MLA (micro lens arrays) must have a matchedF#, which results in complicated and costly lens system designs;

4. The large diameter of the objective lens needed to achieve areasonable size field of view typically results in a rather largeoptical track length which in turn causes the volumetric size of thelight field camera to become large, thus reducing the utility of thecamera and preventing its use in mobile applications;

5. The objective lens system adds well known optical distortions andaberrations, such as barrel distortion, TV distortion, etc. . . . ,which reduce the optical quality of the captured light field and in turndistort the depth and directional information captured by such cameras;and

6. The light field captured by such cameras usually suffers fromunder-sampling and resultant sampling artifacts because the limitedresolution of the sensor, which typically has to be apportioned betweenthe achievable spatial and angular resolution, limits the total numberof directions these light field cameras can capture.

It is therefore an objective of this invention to introduce aspatio-temporal light field camera that overcomes the limitations andweaknesses of the prior art, thus making it feasible to create a lightfield camera that can be embedded in mobile devices and offer the usersof such devices the capability of computational focusing of 2D imagesand the capture of 3D images over a wide angular extent. Additionalobjectives and advantages of this invention will become apparent fromthe following detailed description of a preferred embodiment thereofthat proceeds with reference to the accompanying drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

The invention is illustrated by way of example, and not by way oflimitation, in the Figures of the accompanying drawings in which likereference numerals refer to similar elements.

FIG. 1A illustrates a prior art light field camera array.

FIG. 1B illustrates a prior art plenoptic camera.

FIG. 1C illustrates a prior art prior radiance capture camera.

FIG. 2 illustrates an isometric view of the principle aspects of thespatio-temporal light field camera of this invention.

FIG. 3 illustrates the directional light field capture aspects of thespatio-temporal light field camera of this invention.

FIG. 4A illustrates the angular extent expansion made possible by thetemporal articulation aspects of the spatio-temporal light field cameraof this invention.

FIG. 4B illustrates an exemplary angular temporal articulation of thespatio-temporal light field camera of this invention.

FIG. 5 illustrates the extended angular coverage cross section of thespatio-temporal light field camera of this invention.

FIG. 6 illustrates isometric, top and side views of one embodiment ofthe spatio-temporal light field camera of this invention.

FIG. 7 illustrates an exploded isometric, side and top views of anotherembodiment of the spatio-temporal light field camera of this invention.

FIG. 8A illustrates an exemplary design of the lens element of the microlens array of the spatio-temporal light field camera of this invention.

FIG. 8B illustrates an exemplary embodiment of the cross section of thefull assembly of the spatio-temporal light field camera of thisinvention.

FIG. 8C illustrates a top view of an exemplary embodiment of thespatio-temporal light field camera of this invention.

FIG. 8D illustrates an exemplary embodiment of the spatio-temporal lightfield camera of this invention having an on center micro lens arraylens.

FIG. 8E illustrates an exemplary embodiment of the spatio-temporal lightfield camera of this invention having an offset center micro lens arraylens.

FIG. 9A illustrates an exemplary embodiment of directionaladdressability within one of the spatial pixel groups of thespatio-temporal light field of this invention.

FIG. 9B illustrates an exemplary embodiment of directionaladdressability within one of the spatial pixel group of thespatio-temporal light field of this invention.

FIG. 9C illustrates the curved temporal parallax that can be captured bythe spatio-temporal light field of this invention.

FIG. 10 illustrates an isometric view of an exemplary embodiment of a3D/2D switchable light field camera implemented by tiling a multiplicityof the spatio-temporal light field cameras of this invention.

FIG. 11 illustrates a block diagram explaining the data processing blockdiagram of the spatio-temporal light field of this invention.

FIG. 12 illustrates the light field captured by the spatio-temporallight field camera of this invention in a two dimensional slice acrossits 4-dimension spatial and directional light field space.

FIG. 13 illustrates the principal of networked light field photographyenabled by the spatio-temporal light field camera of this invention

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

An objective of this invention, therefore, is to introduce novel lightfield cameras that are compact enough to readily fit in mobile devices.It is also an objective of this patent to introduce new concepts oflight field photography that emerge from being able to embed the ultracompact light field cameras of this invention in networked mobiledevices.

References in the following detailed description of the presentinvention to “one embodiment” or “an embodiment” mean that a particularfeature, structure, or characteristics described in connection with theembodiment is included in at least one embodiment of the invention. Theappearances of the phrase “in one embodiment” in various places in thisdetailed description are not necessarily all referring to the sameembodiment, and many aspects are applicable to all embodiments.

A new class of light field cameras, referred to as spatio-temporal lightfield cameras, is described herein. Similar to prior art light fieldcameras, such as the plenoptic camera or the radiance camera brieflydescribed earlier, the spatio-temporal light field cameras of thisinvention are also based on the use of high pixel density photo-detectorarray (PDA) device to record the images it captures. Therefore, withinthe context of this invention the term “PDA device” is henceforthintended to mean a photo-detector device that comprises an array ofmicro-scale light detecting pixels. Such a high pixel densityphoto-detector array (PDA) device, hereinafter refer to as simply thePDA device, could either be a charge coupled device (CCD) type of lightsensor or a solid state light (SSL) light sensor such as a CMOS sensoror a light sensor fabricated using III-V material. The electrical outputvalues of the array of pixels of such a PDA device, which would becommensurate with color and intensity of light coupled into the pixel'saperture, would be addressable and collected by a drive circuitrycontained within a CMOS chip (or substrate) upon which the micro-scalepixel detector array is bonded. The size of the pixels comprising thedetector array of the PDA device would typically be in the range ofapproximately 2-10 micron in linear dimension with the typical detectionsurface area of the device being in the range of approximately 0.3-5square centimeter. The electrical output values from the pixels from themicro-scale pixel array of the PDA device are individually addressablespatially, chromatically and temporally either through the drivecircuitry of its CMOS substrate or though an external circuit thatsupports the operation of the PDA device.

The present invention combines the micro pixel array capabilities of thePDA device with passive wafer level optics (WLO) and an articulatedmovement of the entire assembly to create light field cameras that canperform the functionalities of any of the prior art light field camerasdescribed earlier. As used herein, wafer level or wafer means a deviceor matrix of devices having a diameter of at least 2 inches, and morepreferably 4 inches or more. WLO are fabricated monolithically on thewafer from polymer using ultra violet (UV) imprint lithography. Amongprimary advantages of WLO are the ability to fabricate small featuremicro lens arrays (MLA) and the ability to precisely align multiple WLOoptical elements (layers) together and with an optoelectronics devicesuch as a PDA device. The alignment precision that can be achieved by atypical WLO fabrication technique can be less than one micron. Thecombination of the individual pixel addressability of the micro pixelarray of the PDA and the WLO micro lens array (MLA) that can beprecisely aligned with respect to the micro pixel array of the PDAeliminates the need experienced in the prior art for having the bulkyobjective lens in the system, thereby reducing the system volume,complexity and cost simultaneously. In this invention, directionalaspects of the detected light is achieved by the combination of theangular extent achieved the WLO and the articulated movement the entireassembly.

FIG. 2 conceptually illustrates spatio-temporal light field cameras ofthis invention. As illustrated in FIG. 2, the spatio-temporal lightfield cameras of this invention are comprised of the PDA device 210 witha WLO micro lens array (MLA) 220 mounted directly on top of itsdetection surface aperture with the entire assembly being temporallyarticulated around both its x and y axis, preferably by angles withinthe range of ±α_(x) and ±α_(y); respectively. The articulation of thePDA/MLA assembly 230 as illustrated in FIG. 2 would be accomplished byplacing the entire assembly on a 2-axis gimbal whereby the x-axis of thegimbal is temporally actuated by an angle within the range of ±α_(x) andthe y-axis of the gimbal is temporally actuated by an angle within therange of ±α_(y). The x-axis and y-axis temporal articulation provided bythe 2-axis gimbal will cause the directional angle of the light thatimpinges the top surface of the PDA/MLA assembly 230 and detected by thePDA device 210 to be temporally extended by 2α_(x) around the x axis andby 2α_(y) around the y axis beyond the angular extent provided by thelens elements of the MLA 220.

Referring to FIG. 3, associated with each of the micro lens elementscomprising the 2-dimensional array MLA 220 is the group of individuallyaddressable PDA pixels (p₁, p₂, . . . , p_(n)) whereby the lightdetected by each of the pixels in this group of pixels would berefracted from one of the unique directions (d₁, d₂, . . . , d_(n))within the numerical aperture (angular extent) of their associated microlens element. The entire micro-pixel array of the PDA device 210 wouldcomprise a multiplicity of pixel groups (G₁, G₂, . . . , G_(N)), hereinalso referred to as pixel detection groups (or synonymously pixelgroups), whereby each pixel group G_(i) would be associated with one ofthe 2-dimensional array MLA 220 lens elements and collectively the pixelgroups (G₁, G₂, . . . , G_(N)) would then represents the spatialdetection array of the spatio-temporal light field cameras of thisinvention. With the temporal articulation illustrated in FIG. 2 and theone-to-one association of the individual pixels (p₁, p₂, . . . , p_(n))within each pixel group and the detected light directions (d₁, d₂, . . ., d_(n)), it becomes possible for the spatio-temporal light fieldcameras of this invention conceptually illustrated in FIG. 2 to haveassociated with each of its pixel groups G_(i) a multiplicity oftemporally multiplexed directions (d_(1i), d_(2i), . . . , d_(ni)); i=1,2, . . . , each being individually addressable by the temporaladdressing of the individual pixels (p₁, p₂, . . . , p_(n)) within eachof the pixel groups (G₁, G₂, . . . , G_(N)). The multiplicity of PDAdevice pixel groups (G₁, G₂, . . . , G_(N)) associated with the2-dimensional array MLA 220 of FIG. 2 would then represent the spatialarray of the spatio-temporal light field cameras of this invention withthe temporally multiplexed directions (d_(1i), d_(2i), . . . , d_(ni));i=1, 2, . . . , representing the multiplicity of light detectiondirections individually addressable through temporal addressability ofthe pixels (p₁, p₂, . . . , p_(n)) of the PDA device 210 comprising eachpixel modulation group. In other words, the spatio-temporal light fieldcameras of this invention would be able to spatially detect lightthrough addressability of the PDA pixel groups (G₁, G₂, . . . , G_(N))and directionally detect the light from each pixel group in thedirections (d_(1i), d_(2i), . . . , d_(ni)); i=1, 2, . . . , throughtemporal addressability of the pixels (p₁, p₂, . . . , p_(n)) comprisingeach pixel group. Therefore, the spatio-temporal light field cameras ofthe invention illustrated in FIG. 2 would be able to detect light bothspatially and directionally, whereby the light detected by each of thespatial locations that equals the detection area of the PDA each of thepixel groups (G₁, G₂, . . . , G_(N)) is individually addressable throughthe addressability of the pixel groups as well as the directionallyaddressable through the temporal addressability of the individual pixelwithin each pixel group.

The x axis and y axis articulation of the PDA/MLA assembly 230 asillustrated in FIG. 2 will cause its light detection capabilities fromthe directions (d₁, d₂, . . . , d_(n)) to be temporally multiplexed intothe multiplicity of light directions (d_(1i), d_(2i), . . . , d_(ni));i=1, 2, . . . , which extend over the angular extent provided by thelens elements of the MLA 220 plus 2α_(x) in the x direction and by2α_(y) in the y directions. This is illustrated in FIG. 4A which showsthe temporal expansion of the PDA/MLA assembly 230 angular extent alongone articulation axis, for the purpose of illustration. Referring toFIG. 4A, the angle Θ represents the angular extent of one lens elementof the MLA 220 and the angle α represents the composite instantaneousarticulation angle of the lens element as a result of the gimbalarticulation by the angles α_(x)(t) and α_(y)(t) around the x-axis andthe y-axis, respectively. The articulation of PDA/MLA assembly 230 asillustrated in FIG. 2 and explained by FIG. 4A enable the micro-scalepixels within the detection array of the PDA device 210, which areindividually addressable through the PDA drive circuitry, to detectlight both spatially, chromatically and directionally, whereby theangular extent of the directionally detected light is temporallyexpanded by an angle 2α_(x) in the x direction and by an angle 2α_(y) inthe y direction beyond the angular extent Θ (or numerical aperture) ofthe lens elements of the MLA 220. Furthermore, temporal articulation ofthe PDA/MLA assembly 230 of the spatio-temporal light field cameras 200of this invention would temporally increase the detectable number oflight directions (d₁, d₂, . . . , d_(n)) by the ratio of the angularextent expansion in each articulation direction expressed as(Θ+α_(x))(Θ+α_(y))/Θ².

The 2-axis articulation of the PDA/MLA assembly 230 of thespatio-temporal light field 200 of this invention can be in eithertemporally continuous or discrete (stepwise). FIG. 4B illustrates thecomposite temporal articulation angle α(t) of the PDA/MLA assembly 230in one axis, for the purpose of illustration, when the articulation istemporally continuous 410 and when the actuation is temporally discrete420. When the temporal articulation of the spatio-temporal light fieldcameras 200 of this invention is discrete or stepwise (420), the typicalangular step size would preferably be proportional with the ratio of theangular extent Θ of the MLA 220 to spatial resolution the PDA/MLAassembly 230. As illustrated in FIG. 5, the temporal articulation of thePDA/MLA assembly 230 of the spatio-temporal light field cameras of thisinvention would typically be a repetitive (or periodic) and independentaround each of the 2-axis. The repetition periods of the articulation ofthe spatio-temporal light field cameras of this invention wouldtypically be proportional to and synchronized with a predefined outputdata frame duration (for the purpose of reference, the image data from atypical video camera outputted at 60 frames per seconds and is oftenreferred to as 60 Hz frame rate output). The maximum values ±α_(max) ofthe temporal articulation illustrated in FIG. 4B would determine thefull angular extent that can be provided by the spatio-temporal lightfield cameras, which is determined by the value ±(Θ+α_(max)), where theangle the angle Θ represents the angular extent of the lens elements ofthe MLA 220. The periodicity of the x-axis and y-axis articulationcollectively would typically be selected to enable temporal coverage ofthe desired full angular extent of the spatio-temporal light fieldcameras 200 of this invention within the required image frame captureduration. In still image capture (photography) the shutter speed wouldbe the parameter equivalent to the image frame capture duration referredto in the preceding discussion. Meaning that when the spatio-temporallight field cameras 200 of this invention are used in still imagephotography, image frame capture duration will be equivalent to theshutter speed in conventional cameras.

FIG. 5 illustrates the angular coverage cross section 510 of the PDA/MLAassembly 230 of the spatio-temporal light field cameras 200 of thisinvention being comprised of the temporal multiplicity of the angularcoverage cross section 520 of the MLA lens element. Appropriatelyselected temporal articulation α_(x)(t) and α_(y)(t) of the PDA/MLAassembly 230 around its x-axis and y-axis; respectively, will generatethe angular coverage 510 that is comprised of the multiplicity of thetemporally multiplexed angular coverage 520 of the MLA 210 lens element.Depending on the magnitude of the angular articulation α_(x) and α_(y)of the PDA/MLA assembly 230 around their x and y axes, the shape of theangular coverage cross section 510 can be tailored in aspect ratio. Thearticulation rate around the x and y directions would be sufficient toensure that the temporal light detection directions within the angularcoverage 510 have adequate duty cycle within the required image captureframe duration. For example, when the required image capture frameduration is 60 image frames per second, which is typically referred toas 60 Hz image frame rate, each of the light directions within each ofthe temporal angular coverage 520 illustrated in FIG. 5 will need to bedetected once per frame, thus making the articulation rate required togenerate angular coverage illustrated in FIG. 5 to be at least 180 Hzaround either the x or the y axis. In other words, for the angularcoverage example illustrated in FIG. 5 where the size of the temporalangular coverage 510 is three times the size of angular coverage 520 ineach axis, the articulation rate around either the x or the y directionsfor the illustration of FIG. 5 would need to be at least three times theimage capture frame rate. The angular coverage 520 of the MLA lenselement can be either overlapping or non-overlapping. In general thearticulation rate of the PDA/MLA assembly 230 around either the x or yaxis would have to be at least equal to the image capture frame ratemultiplied by a factor that equals the ratio of the size (in degrees) ofthe angular coverage 510 along each axis to the size (in degrees) of theangular coverage 520 along the same axis.

Referring to FIG. 5, with the temporal articulation of the PDA/MLAassembly 230 of the spatio-temporal light field cameras 200 of thisinvention having the angular coverage 520 and comprising themultiplicity of light detection directions corresponding to themultiplicity of spatial pixels groups comprising the PDA device 210, anew set of light detection directions would be continuously added assome drop off temporally in a pipeline fashion until the full angularextent 510 of the spatio-temporal light field cameras 200 of thisinvention is fully covered. At any given instant the full aperture ofthe PDA/MLA assembly 230 would be utilized to accumulate the light fromany given direction as that direction remains temporally within thecoverage of the articulated aperture 510. As a result of thisspatio-temporal pipelining of the multiplicity of the detection of lightdirections, the response time of the spatio-temporal light field cameras200 of this invention can be made to be commensurate with the imagecapture frame rate with minimal latency. The time duration a given lightcapture direction remains within the angular coverage 520 woulddetermine the integration time available for the capture of the lightintensity entering the spatio-temporal light field cameras 200 from thatdirection. As a result, unless compensated, the light directions withinthe peripheral area of the full angular coverage 510 could have lessintegration time, and therefore intensity, than the interior region ofthe angular coverage 520. This intensity edge tapering effect would besomewhat similar to the Fresnel losses typically encountered at the edgeof an optical system except in the case of the spatio-temporal lightfield cameras 200 of this invention, such an effect can be compensatedby appropriate selection of the rate of the temporal articulation of thePDA/MLA assembly 230 of the spatio-temporal light field cameras 200 ofthis invention.

As an alternative, the temporal angular articulation may be by way of asort of step and repeat process wherein the temporal angulararticulation is in angular steps. This equalizes the time of exposure ofeach pixel to light from overall field of view. As a still furtheralternative, the temporal angular articulation may be by way of asinusoidal variation, with a pause at the maximum articulation positionsso that the exposure times of the pixel groups at the maximumarticulation positions is increased.

One embodiment of this invention, herein referred to as 600, isillustrated in FIG. 6, which includes an isometric, top view and sideview illustrations of this embodiment. As illustrated in FIG. 6, thespatio-temporal light field cameras of this invention are realized bybonding the PDA/MLA assembly 230 (depicted in FIG. 2) on the topside ofthe 2-axis gimbal assembly 620 which is fabricated using multiplesilicon substrate layers; namely, a hinge layer 621, a spacer layer 628and a base layer 630. As illustrated in FIG. 6, the hinge layer 621 ofthe 2-axis gimbal 620 is comprised of an outer frame 622, an inner ring623 and the inner segment 625 upon which PDA/MLA assembly 230 would bebonded (625 is hereinafter also referred to synonymously as the bondingpad 625). The gaps between the outer frame 622, the inner ring 623 andthe inner segment 625 would be etched using standard semiconductorlithography techniques. The inner segment 625 is physically connectedalong the x-axis to the inner ring 623 by two silicon bridges 622, eachtypically approximately in the range of 0.3-0.5 mm wide, which would actas the x-axis hinge and would also to define the neutral x-axis positionof the gimbal and act as a mechanical resistance spring for the x-axisarticulation. The inner ring 623 is connected along the y-axis to theouter frame 622 by two silicon bridges 626, each typically approximatelyin the range of 0.3-0.5 mm wide, which would act as the y-axis hinge andwould also to define the neutral y-axis position of the gimbal and actas a mechanical resistance spring for the y-axis articulation. The twopairs of silicon hinges 624 and 626 constitute the pivot points of the2-axis gimbal around which the x and y articulation would be performed.The interior segment 625 of the hinge layer 621 of the gimbal assembly620 contains a multiplicity of contact pads to which the PDA/MLAassembly 230 is bonded using standard soldering techniques such as flipchip solder balls, thus making the inner segment 625 become the bondingpad upon which PDA/MLA assembly 230 would be bonded. Embedded within theinterior segment 625 of the hinge layer 621 of the gimbal assembly 620are a multiplicity of metal rails which connect a set of contact pads onthe topside of the interior segment 625 to a set of device contact pads627 placed along the periphery of the outer frame 622 via the x-axis andy-axis hinge bridge areas 624 and 626. The set of contact pads on thetopside of the interior segment 625 are the contact pads that wouldprovide electrical and physical contact to the backside of the PDA/MLAassembly 230.

Referring to the side view illustration of FIG. 6, the PDA/MLA assembly230 is shown bonded to the topside of the interior segment 625. Asexplained earlier, this would be both an electrical and physical contactbonding between the contact pads on the topside of the interior segment625 and the contact pad at the backside of the PDA/MLA assembly 610using solder or eutectic ball grid array type bonding. Also illustratedin FIG. 6 side view is the spacer layer 628 which would be bonded atwafer level with the base layer 630 topside and with the hinge layerbackside using BenzoCycloButene (BCB) polymer adhesive bonding or thelike. The height (or thickness) of the spacer layer 626 would beselected to accommodate the vertical displacement of the corner of thehinged interior segment 625 together with the bonded PDA/MLA assembly610 at the maximum actuation angle. For example, if the diagonal of thebonding pad 625 measures 5 mm and the maximum articulation angle at thecorner is 15°, then the thickness of the spacer layer 626 should measureapproximately 0.65 mm in order accommodate the vertical displacement ofthe corner of the bonding pad 625 at the maximum articulation angle.

Referring to the side view illustration of FIG. 6, the articulation ofthe bonding pad 625 together with the bonded PDA/MLA assembly 230 wouldbe accomplished using a set of electromagnets 635 placed at the fourcorners of the backside of the hinged bonding pad 625, and a set ofpermanent magnets 636 placed on the topside of base layer 630 inalignment with the four corners of the backside of the hinged bondingpad 625. The electromagnets 635 would be a coil having a metal coreformed at wafer level using multilayer imprint lithography on thebackside of the hinged bonding pad 625. The permanent magnets 636 wouldbe a thin magnetic strip typically of neodymium magnet (Nd₂Fe₁₄B) or thelike. Articulation of the hinged bonding pad 625 together with thebonded PDA/MLA assembly 230 as described earlier would be accomplishedby driving the set of electromagnets 635 with an electrical signalhaving the appropriate temporal amplitude variation to affect theappropriate temporal variation in the magnetic attraction between theset of electromagnets 635 and permanent magnets 636 that would cause thehinged bonding pad 625 together with the bonded PDA/MLA assembly 230 tobe temporally articulated as described earlier. The drive electricalsignals to the set of electromagnets 635, which are generated either bythe PDA device 210 or by an external support device, are supplied to theset of electromagnets 635 via the metal rails and contacts incorporatedin the hinged interior segment 625 described earlier, and aresynchronous with the image frame capture duration (rate) performed bythe PDA device 210 to the extent that will enable the desireddirectional detection of the light on the pixel array of the PDA device210. The temporal variation of the drive electrical signals to the setof electromagnets 635 would be selected to enable the temporal angulararticulation of the hinged bonding pad 625 together with the bondedPDA/MLA assembly 230 around both of their x-axis and y-axis asillustrated in FIG. 6. Depending on the thickness of the siliconsubstrate of the hinge layer 621 and the selected width of the siliconhinges 624 and 626, the maximum value ±α_(max) of the temporal angulararticulation α(t) illustrated in FIG. 4B that can be achieved byembodiment 600 of this invention would typically be in the range from±15° to ±17°.

The drive electrical signals to the set of electromagnets 635, which areeither generated by the PDA device 210 or an external support device,are supplied to the set of electromagnets 635 via the metal rails andcontacts incorporated in the hinged interior segment 625 describedearlier, would be comprised of a base component and a correctioncomponent. The base component of the drive electrical signals to the setof electromagnets 635 would represent a nominal value and a correctioncomponent would be derived from an angular articulation error valuegenerated by a set of four sensors positioned on the backside of thehinged interior segment 625 in alignment with the hinges 624 and 626.These sensors would be an array of infrared (IR) detectors placed on thebackside of the hinged interior segment 625 in alignment with four IRemitters placed on the topside of the base layer 630. The output valuesthese four IR detector arrays will be routed to either the PDA device oran external support device, again via the metal rails and contactsincorporated in the hinged interior segment 625 described earlier, andare used to compute an estimate of the error between the derived and theactual articulation angle which will be incorporated as a correction tothe drive signals provided by either the PDA device or an externalsupport device to the set of electromagnets 635. The sensors positionedon the backside of the hinged interior segment 625 could also bemicro-scale gyros properly aligned to detect the actuation angle alongeach of the 2-axis of the gimbal.

Another embodiment of this invention is illustrated in FIG. 7, hereinreferred to as 700. FIG. 7 includes isometric views and side viewillustrations of this embodiment. As illustrated in FIG. 7, theembodiment 700 of this invention is comprised of the 2-axis gimbal 720with the PDA/MLA assembly 230 bonded on top of it. FIG. 7 also shows anexploded isometric illustration of the embodiment 700 that shows theconstituent layers of the 2-axis gimbal 720 of this embodiment. Asillustrated in FIG. 7, the spatio-temporal light field cameras of thisembodiment are realized by bonding the PDA/MLA assembly 230 (depicted inFIG. 2) on the topside of the 2-axis gimbal assembly 720 which isfabricated using multiple silicon substrate layers; namely, a pad layer721, a spring layer 725 and a base layer 730. The topside of the padlayer 721 incorporates multiplicity of contact pads to which the PDA/MLAassembly 230 is to be bonded using standard soldering techniques such asflip chip solder balls, thus making the topside of the pad layer 721being the bonding pad 723 upon which PDA/MLA assembly 230 is bonded. Thebackside of the pad layer 721 incorporates the spherical pivot 735 whichwould be formed by embossing polycarbonate polymer on the backside ofthe hinged pad layer 721 at the wafer level using UV imprint lithographyor the like. The pad layer 712 together with the spherical pivot 735embossed on its backside will be referred to as hinged pad 721/735. Thetopside of the base layer 730 incorporates the spherical socket 736which would be formed by embossing of polycarbonate polymer on thetopside of the base layer 730 at the wafer. The base layer 730 togetherwith the spherical socket 736 embossed on its topside will be referredto as the pedestal 730/736. The surface curvature the spherical pivot735 incorporated on the backside of the pad layer 721 and the sphericalsocket 736 incorporated on the topside of the base layer 730 will bematched in order to allow the hinged pad 721/735 to make it 2-axisarticulated pad when placed on top of the pedestal 730/736. Although theembossed surfaces of the spherical pivot 735 and socket 736 will be ofoptical quality in terms of surface roughness in the order of a few nmRMS, possible friction between the two surfaces due to the articulationmovement would be reduced by coating the surfaces of the spherical pivot735 and socket 736 with a thin layer (50-100 nm) of graphite.

The hinged pad 721/735 is retained in place within the surface curvatureof the pedestal 730/736 by the spring layer 725 which contains at eachof its four corners a single spiral shaped spring 726 that is etchedinto the spring layer 725. As illustrated in FIG. 7 exploded viewisometric, the inner end of each of the four spiral shaped springsincorporates an inner bonding pad 727 which corresponds to an identicalbonding pad 722 located at the backside of the pad layer 721. Embeddedwithin the spiral shaped springs 726 are multiple metal rails which areused to route the electrical interface signals from the inner bondingpad 727 to a set of contact pads 728 located at the peripheral edge ofthe backside of the spring layer 725. The edge contacts 728 on thebackside of the outer end of the spring layer 725 correspond to amatching set of bonding pads 729 that are located at the peripheral edgeof the base layer 730. The edge contacts on the topside of the baselayer 730 are connected via metal rails embedded within the base layerto a set of device contact pads 731 that are located on the backside ofthe base layer 730. In the final assembly of the embodiment 700 of thisinvention, illustrated in the side view of FIG. 7, the four spiralshaped springs 726 will be expanded when the backside of bonding pads726 of the spring layer 725 are bonded to the topside bonding pads 729of the base layer 730 and the inner bonding pad 727 of the spiral spring726 is bonded to the corresponding bonding pad 722 on the backside ofthe pad layer 721. When the spring layer 725 is bonded to the backsideof the pad layer 721 and to the topside of the base layer 730 spiralsprings 726 as just explained, the four spiral springs become fullyexpanded and in that full expanded configuration they serve the multiplepurposes of: (1) creating a spring load resistance needed to keep thespherical pivot 735 retained within the spherical socket 736; (2)creating the mechanical balance needed for sustaining the neutralposition of the hinged pad 721/735; and (3) routing the electricalinterface signals from the device contact pads 731 to the contact pad723 of the PDA/MLA assembly 230. Referring to the side view illustrationof FIG. 7, the PDA/MLA assembly 230 is shown bonded to the topsidecontact pad 723 of the pad layer 721. This would be both an electricaland physical contact bonding between the contact pads 723 and thecontact pad at the backside of the PDA/MLA assembly 230 using solder oreutectic ball grid array type bonding. In the operational configurationthe full device assembly 700 would be bonded using the contact pad 731located on the backside of the base layer to a substrate or printedcircuit board using solder or eutectic ball grid array type bonding.

Also illustrated in FIG. 7 side view is the extended height of thespherical socket 736 which would be selected to accommodate the verticaldisplacement of the corner of the hinged pad 721/735 together with thebonded PDA/MLA assembly 230 at the maximum actuation angle. For example,if the diagonal of the hinged pad 721/735 together with the bondedPDA/MLA assembly 230 measures 5 mm and the maximum actuation angle atthe corner is ±30°, then the thickness of the extended height of thespherical socket 736 should measure approximately 1.25 mm in order toaccommodate the vertical displacement of the corner of the of the hingedpad 721/735 together with the bonded PDA/MLA assembly 710 at the maximumactuation angle.

The actuation of the hinged pad 721 together with the bonded PDA/MLAassembly 230 would be accomplished using a set of electromagnetsembedded within the spherical pivot 735 and a set of permanent magnetsembedded within the spherical socket 736. The actuation electrical drivesignal would be routed to electromagnets embedded within the sphericalpivot 735 in order to affect the actuation movement described in theearlier paragraphs. The base component of the actuation electrical drivesignals to the electromagnets embedded within the spherical pivot 735would represent a nominal value and a correction component that would bederived from an angular articulation error value generated by a set offour sensors positioned on the backside of the hinged pad 721. Thesesensors are an array of infrared (IR) detectors placed on the backsideof the hinged pad 721 in alignment with four IR emitters placed on thetopside of the base layer 730. The output values these four IR detectorarrays will be routed to the PDA device or an external support device,again via the metal rails and contacts incorporated in the hinged pad721 described earlier, and used to compute an estimate of the errorbetween the derived and the actual articulation angle which will beincorporated as a correction to the drive signals provided by the PDAdevice to the set of electromagnets embedded within the spherical pivot735. The sensors positioned on the backside of the hinged pad 721 couldalso be micro-scale gyros, implemented using micro-electro mechanicalsystems (MEMS) or piezoelectric micro gyroscopes, properly aligned todetect the actuation angle along each of the 2-axis of the gimbal.

The permanent magnets embedded within the spherical socket 736 would bethin magnetic rods or wires, typically of neodymium magnet (Nd₂Fe₁₄B) orthe like, and would be shaped to provide a uniform magnetic field acrossthe curved cavity of the spherical socket 736. Actuation of the hingedpad 721 together with the bonded PDA/MLA assembly 230 as describedearlier would be accomplished by driving the set of electromagnetsembedded within the spherical pivot 735 with an electrical signal havingthe appropriate temporal amplitude variation to affect the appropriatetemporal variation in the magnetic attraction between the set ofelectromagnets embedded within the spherical pivot 735 and permanentmagnets embedded within the spherical socket 736 that would cause of thehinged pad 721 together with the bonded PDA/MLA assembly 230 to betemporally articulated as described earlier. The drive electricalsignals to the set of electromagnets embedded within the spherical pivot735, which are generated by either the PDA device or an external supportdevice and routed via the metal rails and contacts incorporated thehinged pad 721 described earlier, would be made synchronous with theimage frame capture duration (rate) performed by the PDA device 210 tothe extent that will enable the desired directional detection of thelight on the pixel array of the PDA device 210. The temporal variationof the drive electrical signals for the set of electromagnets embeddedwithin the spherical pivot 735 would be selected to enable the temporalangular articulation of the hinged pad 721 together with the bondedPDA/MLA assembly 230 along both of their x-axis and y-axis asillustrated in FIG. 6. Depending on the extended height of the sphericalsocket 736 which governs the maximum vertical displacement of the cornerof the hinged pad 721 together with the bonded PDA/MLA assembly 230, themaximum value ±α_(max) of the temporal angular articulation α(t)illustrated in FIG. 6 that can be achieved by the embodiment 700 of thisinvention would typically be in the range from ±30° to ±35°.

A person skilled in the art would know that the gimbal actuators of theembodiments 600 and 700 of this invention described in the previousparagraphs can be implemented to achieve substantially the sameobjective by exchanging the positions of the electromagnets and thepermanent magnets. Furthermore, a person skilled in the art would alsoknow that the gimbal actuators of the embodiments 600 and 700 of thisinvention described in the previous paragraphs can be implemented toachieve substantially the same objective using actuation drive methodsother than the electromagnet based method described in the previousparagraphs.

The two embodiments 600 and 700 of the spatio-temporal light fieldcameras of this invention differ mainly in the maximum value α_(max) ofthe temporal angular articulation α(t) each can achieve and in the outerarea each embodiment extends beyond the boundary of the PDA/MLA assembly230. First, as illustrated in FIG. 7, in the embodiment 700 of thisinvention the 2-axis gimbal is fully accommodated within the footprintarea of the PDA/MLA assembly 230 (hereinafter refer to a zero-edgefeature) while as illustrated in FIG. 6 in the embodiment 600 of thisinvention the 2-axis gimbal is accommodated at the outer periphery ofthe PDA/MLA assembly 230 outer boundary. Second, the maximum valueα_(max) of the temporal angular articulation α(t) embodiment 700 canachieve could possibly be twice as large as what could be provided forembodiment 600. Of course the larger maximum value α_(max) of thetemporal angular articulation α(t) that can be accomplished by theembodiment 700 comes at the expense of requiring larger vertical heightthan the embodiment 600. The zero-edge feature of the embodiment 700makes it more suitable for being tiled to create a large aperture lightfield camera (as will be explained in later paragraphs) while the lowprofile (low height) feature of the embodiment 600 makes it moresuitable for creating an ultra compact light field camera for mobileapplications.

FIG. 8A shows an exemplary embodiment of one element of the MLA 220 andits associated pixel group G_(i) of the PDA device 210 that can be usedwithin the context of the present invention. Referring to FIG. 8A, asexplained earlier the light detected by each individual pixel within apixel group G_(i) reaches the surface of the PDA device 210 through theaperture of a micro lens element that comprises the three opticalelements 810, 820 and 830. Each light bundle that traverses a specificdirection and impinges the aperture of the MLA 220 within a angularextent δΘ, which is referred to herein as the angular resolution of thelight field cameras of this invention, would be collimated by the MLA220 micro lens system 810, 820 and 830 and reach one of the individualpixels within a pixel group G_(i) of the PDA device 210. In essence themicro lens system illustrated in FIG. 8A comprising the optical elements810, 820 and 830 would map the incident light from the multiplicity ofdirections within the light field defined by an exemplary Θ=±15° angularextent of the MLA 220 micro lens system 810, 820 and 830 to theindividual pixels comprising the two dimensional pixels of the pixelgroup G_(i) (see FIG. 3). It should be noted that in order to capturethe full depth of the light field, the MLA 220 micro lens system 810,820 and 830 is designed to have an infinite depth of focus (i.e.,focused on infinity).

To a person skilled in the art, the exemplary embodiment of one elementof the MLA 220, illustrated in FIG. 8A comprising micro lens system 810,820 and 830, would be known to as a Fourier optical system because itmaps the directional aspects of the light field that impinges theaperture of the MLA 220 within the angular extent Θ, which in the caseof exemplary embodiment of FIG. 8A is an exemplary ±15°, into the twodimensional spatial array of the pixels comprising the pixel group G_(i)of the PDA device 210 that is associated with that MLA 220 micro lenselement. In essence the MLA 220 of the light field cameras of thisinvention is an array of micro-scale Fourier lenses, or a Fourier microlens array. As explained earlier, the MLA 220 would be comprised of amultiplicity of the elements each being comprised of exemplary lenssystem 810, 820 and 830 whereby each of the MLA 220 would be associatedwith one of pixels within a pixel group G_(i) comprising the PDA device210. With each one of the MLA 220 micro lens elements, which could becomprised of the exemplary lens system 810, 820 and 830, mapping thelight field that impinges its aperture within the angular extent Θ intothe two dimensional array of the pixel group G_(i) associated with it,the PDA/MLA assembly 230 of the spatio-temporal light field cameras ofthis invention would be able to capture light field that impinges on itsaperture within the angular extent Θ with a directional resolution thatequals the total number of pixels (p, p₂, . . . , p_(n)) comprising eachof the pixel array groups G_(i) and a spatial resolution that equals thetotal number of pixel groups (G₁, G₂, . . . , G_(N)) comprising the PDAdevice 210. In other words the PDA/MLA assembly 230 of thespatio-temporal light field cameras of this invention would be able tocapture a multiplicity of views, that equals to the number n of pixelscomprising each of the pixel groups G_(i), of the light field thatimpinges its aperture within a solid angle Θ with the spatial resolutionof each of the views it captures being equal to the number N of pixelgroups comprising the PDA device 210. As explained earlier, when thePDA/MLA assembly 230 is used within the context of the either of theembodiments 600 or 700 having a maximum temporal articulation ofα_(max)=±15°, for example, the angular extent of this the exemplaryembodiment of the light field cameras of the this invention would be±(Θ+α_(max))=±30° and the number of views it can capture would be

${\left\lbrack \frac{\left( {\Theta + \alpha_{\max}} \right)^{2}}{\Theta^{2}} \right\rbrack n} = {4n}$

views each being captured with a spatial resolution N.

The angular extent Θ of the MLA 220 exemplary embodiment comprisingmicro lens system 810, 820 and 830 of FIG. 8A can be made either largeror smaller than the ±15° through appropriate design selection of therefracting surfaces of the micro lens system 810, 820 and 830 or byincreasing or decreasing the number of its optical elements. It shouldbe noted, however, that for a given directional resolution, which isherein defined as the number of directional views determined by thenumber of pixels within the pixel modulation group G_(i), changing theangular extent Θ of the MLA 220 micro lens system would result in achange in the angular separation between the directional views, which isherein defined as the angular resolution, detected by the PDA/MLAassembly 230 of the spatio-temporal light field cameras of thisinvention. For example with the Θ=±15° angular extent of the previousexemplary embodiment, if the pixel group G_(i) comprises (16×16) pixels,then the angular separation (or angular resolution) between thedirectional light beam detected by the PDA/MLA assembly 230 of thespatio-temporal light field cameras of this invention would beapproximately δΘ=1.875°. This same angular resolution value of δΘ=1.785°can also be achieved by reducing the angular extent of the MLA 220 microlens system to Θ=±7.5° and the number of pixels comprising the pixelgroup G_(i) to (8×8) pixels. In general using a higher optical apertureF/# (i.e., smaller value of the angular extent Θ) for the MLA 220 microlens system would allow achieving a given angular resolution value usinga smaller pixel group G_(i) size, which in turn would result in theavailability of more pixels within a given pixel resolution of the PDAdevice 210 to create more of the pixel groups G_(i) and consequentlyallow higher spatial resolution.

This design tradeoff allows selecting the appropriate balance betweenthe optical aperture F/# of the MLA 220 micro lens system designparameters and spatial resolution that can be achieved by the PDA/MLAassembly 230. On the other hand, when the optical aperture F/# of theMLA 220 micro lens system is increased to increase the spatialresolution, as just explained, the angular extent that can be achievedby the PDA/MLA 220 of the spatio-temporal light field cameras of thisinvention would be reduced. At this point the maximum value α_(max) ofthe temporal angular articulation α(t) of this invention will become apart of the design tradeoff to recover the angular extent sacrificed infavor of increasing the spatial resolution. In the previous example whenthe maximum value α_(max) of the articulation angle is selected to beα_(max)=±7.5°, the spatio-temporal light field cameras of this inventionwill be able to achieve an full angular extent of (α_(max)+Θ)=±15° usingthe pixel group G_(i) of (8×8) pixels. In essence for a given angularresolution value of δΘ, the maximum value of the articulation angleα_(max) comes into the tradeoff as a parameter that can be used eitherto increase the angular extent or the spatial resolution that can beachieved by the spatio-temporal light field cameras of this invention,or a combination of the angular extent and the spatial resolution. Ofcourse suitable actuators for the angular articulation are notnecessarily limited to electro-magnetic actuators, but other types ofactuators may be used if desired. By way of example, particularly if theangular extent of the PDA/MLA assembly 230 is adequate and the angulararticulation is used to increase the angular resolution, then the amountof angular articulation required will be quite small, namely less thanthe angular resolution without articulation. Consequently electromechanical actuators that have small deflections can be used, such aspiezo-electric actuators. Such actuators can be highly reliable,efficient, low cost, fast and easily controlled. They also provide fixedpositions versus voltage applied, to be compared with forcesproportional to current without a reference position provided by voicecoil type of electromagnetic actuators, which may eliminate the need forphysical gimbals, thereby further simplifying the mechanical assembly.

FIG. 8B shows an exemplary embodiment of the full assembly of thePDA/MLA assembly 230 of the spatio-temporal light field cameras of thisinvention. The multiplicity of the micro lens elements 810, 820 and 830are fabricated to form the micro lens arrays layers 840, 850 and 860which would be precisely aligned relative to each other and relative tothe associated arrays of the PDA device 210 pixel groups (G₁, G₂, . . ., G_(N)) at the wafer level using semiconductor aligners, which cantypically achieve wafer-to-wafer alignment accuracy below 0.25 micron.The exemplary embodiment illustrated in FIG. 8B also includes the PDAdevice 210 and the cover layer 870, which would typically be a glasslayer that is incorporated as a protective encapsulation of the PDAdevice 210. The design of the micro lens elements 810, 820 and 830 wouldtake into account the thickness and optical characteristics of the PDAcover glass 870 in order to make the image be at the surface of the PDAdevice 210. The exemplary embodiment of FIG. 8B illustrates the fullassembly of the PDA/MLA 230 that can be used within the context of theembodiments 600 or 700 of the spatio-temporal light field cameras ofthis invention. Using the exemplary embodiment of FIG. 8B, the typicaltotal thickness of the embodiments 600 and 700 of the spatio-temporallight field cameras of this invention would be less than 5 mm. Suchcompactness of the light field cameras of this invention is not possiblyachievable by any of the light field camera techniques of the prior art.It should be noted that as shown in FIG. 8C, which illustrate a top viewof one quadrant of the MLA 220 comprising the micro lens arrays layers840, 850 and 860, the center of the each of the MLA 220 lens elementsare aligned with the center of their respective pixel groups G_(i) inorder to provide full angular coverage of the light field across thespatial extent of the MLA 220 without any directional coverage gaps.

Another embodiment of the spatio-temporal light field cameras of thisinvention is illustrated in FIG. 8D which shows a top view of onequadrant of the MLA 220. In comparison with the MLA 220 micro lenselement illustrated in FIG. 8C in which the lens array elements of theMLA 220 are truncated along both the x-axis and y-axis to have theircross section match the dimensional aspects of their respective pixelgroup G_(i) and their optical centers are aligned with the centers oftheir respective pixel group G_(i), the optical centers of the MLA 220lens elements of the embodiment illustrated in FIG. 8D is graduallyoffset from the centers of their respective pixel group G_(i). Asillustrated in FIG. 8D, the lens element at the center of the MLA 220would be aligned with the center of its respective pixel group but thelens elements away from the center of MLA 220 have their centers offsetfrom the center of their respective pixel group with such an offsetgradually increasing for lens elements further away from the center ofthe MLA 220. The virtue of this embodiment is that it enables yetanother aspect of design tradeoff between the achievable angularresolution, directional resolution and spatial resolution of thespatio-temporal light field cameras of this invention. As explainedearlier, the achievable angular resolution 60 would be determined by theangular extent Θ of the lens elements of the MLA 220 and the directionalresolution defined by the size of the pixel group G_(i). For a givenvalue of the angular extent Θ, the angular resolution δΘ decreases andthe directional resolution increases with the increase in the size ofthe pixel group G_(i). Therefore, for a given size PDA device 210, interms of the number of available pixels, increasing the achievabledirectional resolution would be at the expense of decreasing theachievable spatial resolution. In order to increase the directionalresolution without decreasing the spatial resolution one would have toreduce the angular extent of the MLA 220, which in turn would reduce theoverall angular extent of the light field cameras. The embodiment of thespatio-temporal light field cameras of this invention illustrated inFIG. 8D offers another alternative of compensating for the resultantreduction in the angular extent Θ of the elements of the MLA 220, infavor of increasing the directional resolution, by gradually increasingthe inclination of the field of view of the elements of the MLA 220 inorder to achieve a larger overall field of view for the cameras.Increasing the inclination of the field of view of the elements of theMLA 220 is achieved, as illustrated in FIG. 8D, by gradually offsettingthe centers of the MLA 220 lens elements from the centers of theirrespective pixel group G_(i). However, the gradual increase in theinclination of the field of view of the elements of the MLA 220, thatresults from the gradual offset of the centers of the MLA 220 elementsas illustrated in FIG. 8D, would result in gradually truncateddirectional coverage gaps from the center to the edge of the PDA device210. On the other hand, such resultant directional coverage truncationgaps will be filled in by the temporal angular articulation of thespatio-temporal light field cameras of this invention. With the designtradeoff offered by the embodiment illustrated in FIG. 8D, therefore, itbecomes possible to achieve higher directional resolution withoutsacrificing either the spatial resolution or the overall angular extentof the spatio-temporal light field cameras of this invention, especiallygiven the angular extent and directional resolution expansion that canbe realized by the angular temporal articulation of the spatio-temporallight field cameras of this invention.

It should be noted that although in the exemplary embodiment of the MLA220 lens element illustrated in FIG. 8A the detection surface of the PDAdevice 210 is shown placed at the focal plane of the MLA 220 lenselement, in another embodiment of this invention that is illustrated inFIG. 8E, the detection surface of the PDA device 210 is alternatively beplaced a further distance away from the focal plane of the MLA 220 lenselement. Furthermore, the distance between the detection surface of thePDA device 210 and the MLA 220 lens element can be also made to beadjustable, in the order of a few microns, by placing the MLA 220assembly on a z-axis electro-mechanical actuator that would be used tovary the distance between the detection surface of the PDA device 210and the MLA 220 lens element within a set range. Such a z-axiselectro-mechanical actuator would be similar to that used as a focusingmechanism in conventional mobile cameras. As illustrated in FIG. 8E,when the distance between the detection surface of the PDA device 210and the MLA 220 lens element is further away from the focal plane of theMLA 220 lens element, the sub-images formed by each of the MLA 220 lenselements on the detection surface of the PDA device 210 will becomeblurred (or de-focused), thus causing the light relayed by the MLA 220lens element from one direction to be spread across multiple pixelswithin the corresponding pixel group G_(i) and the sub-image formed byeach of the lens elements of the MLA 220 on the detection surface of thePDA device 210 to be spread across the pixels expanded beyond theboundaries of their respective the pixel groups G_(i). As a result,therefore, the directional information of the light relayed to the PDAdevice 210 by a given lens element of the MLA 220 will be spread acrossa larger number of pixels of the PDA device 210 than that of thecorresponding pixel groups G_(i) of that lens element; which in turnmeans that the directional information of the light relayed by the lenselements of the MLA 220 would be recorded by the PDA device 210 withhigher directional resolution; or alternatively a smaller value of theangular extent δΘ (i.e., higher angular resolution). In effect with thisembodiment of the spatio-temporal light field cameras of this inventionthe plurality of pixels comprising each of the PDA device 220 pixelgroups (G₁, G₂, . . . , G_(N)) associated with the 2-dimensional arrayMLA 220 would be shared collectively in recording the light field withhigher angular and directional resolutions. The higher angularresolution is achieved because the light relayed by the MLA 220 lenselement from one direction is spread across multiple pixels (asillustrated in FIG. 8E), rather than being focused on only one pixel (asillustrated in FIG. 8A), thus causing the light within the angularextent Θ of each of the MLA 220 elements relayed from a specificdirection to be sampled by a larger number of the PDA device 210 pixels;therefore achieving a smaller value of angular extent δΘ. The higherdirectional resolution is achieved because the sub-images formed by eachof the lens elements of the MLA 220 on the detection surface of the PDAdevice 210 would be spread (blurred or defocused) across pixels expandedbeyond the boundaries of their respective the pixel groups G_(i), thuscausing the light within the angular extent Θ of each of the MLA 220elements to be sampled with larger number of the PDA device 210 pixels,which means larger number of light directions (or views) would bedetected. The information recorded by each of the pixels comprising eachof the pixel groups G_(i) would therefore be, depending on the selecteddistance between the PDA device 210 and the MLA 220, a known weightedsum (defined by the optical transfer function of the MLA 220 lenselement) of the light field directional information relayed by amultiplicity of lens elements of the MLA 220 which can becomputationally resolved. The tradeoff implied by the sub-imageexpansion caused by the increasing the distance between the PDA device210 and the MLA 220 would be an increase in the computational resourcesneeded to resolve the light field information recorded by thespatio-temporal light field cameras of this invention. With thisembodiment the angular and directional resolutions of thespatio-temporal light field cameras of this invention can be selected ata given value; and thus also the computational resources needed toresolve the captured light field, by either a prior design selection ofa set distance between the PDA device 210 and the MLA 220, or byoperational mode adjustments of the distance between the PDA device 210and the MLA 220 using the z-axis actuator mentioned earlier. It shouldbe noted that in lieu of increasing the angular and/or directionalresolutions of the spatio-temporal light field cameras of thisinvention, the embodiment of FIG. 8E can be used to reduce the size ofthe pixel groups G_(i) needed to achieve a required directional orangular resolutions; thus making more of the pixels of a given size PDAdevice 210 become available for achieving higher spatial resolution. Inessence, therefore, the embodiment of FIG. 8E when taken together withthe temporal angular articulation of the spatio-temporal light fieldcameras of this invention would allow the angular and directionalresolutions added by the temporal angular articulation to enable higherspatial resolution.

It should be noted that the spatio-temporal light field cameras of thisinvention differs from prior art light field cameras described earlierin many very important aspects, the most relevant are discussed herein.The first being that unlike all other prior art, the spatio-temporallight field cameras of this invention do not use an objective lens anddo not rely on the principle of sampling the aperture of that objectivelens, instead the spatio-temporal light field cameras of this inventionuse the array 220 of micro-size Fourier lens system, such as that ofexemplary embodiments 800, to sample the entire light field within theangular extent of the light field camera with a maximum depth of fieldsince the individual elements of the micro-size Fourier lens system (theMLA 220) would be focused at infinity. This approach of using an MLA 220of micro-size Fourier lens system, such as that of exemplary embodiments800, to directly sample the light field makes the required optical tracklength, which is typically commensurate with the lens aperturediameters, be rather small and typically in the order of fewmillimeters, unlike prior art light field cameras described earlierwhich typically have an optical track length in the range of 10-15centimeter or greater. This approach, therefore, makes thespatio-temporal light field cameras of this invention realize anunprecedented volumetric advantage over all other prior art light fieldcameras. As explained earlier, the prior art light field cameras havethe problem that their optical track length, and hence their volumetricaspects, would be rather large due to the fact that their objective lensdiameter is large; a factor that also increases their opticalaberration. The spatio-temporal light field cameras of this inventionavoid both of these problems by using an array of smaller diametermicro-lenses array that is temporally articulated.

The second distinctive aspect of the spatio-temporal light field camerasof this invention is that unlike prior art light field cameras describedearlier in which the angular extent is solely determined by theirobjective lens, the angular extent of the spatio-temporal light fieldcameras of this invention is determined by the combination of theangular extent of the MLA 220 plus the angular temporal articulation ofthe PDA/MLA assembly 230. That distinction makes the spatio-temporallight field cameras of this invention have several advantages that arenot shared with the prior art light field cameras described earlier. Thefirst of these advantages, which was described earlier, is that theangular articulation angle can be used either to increase the angularextent, the directional resolution, the angular resolution or thespatial resolution that can be achieved by the spatio-temporal lightfield cameras of this invention. This is a key advantage because itmakes it possible to realize a much higher spatial resolution per viewfrom a given PDA device than prior art light field cameras. The secondof these advantages stems from the fact that the angular extent of theprior art light field cameras described earlier is solely determined bytheir objective lens, increasing the size of the angular extent wouldtypically require decreasing the size of the aperture stop, which inturn would result in a reduction in the light entering the camera and aconsequent increase in the signal to noise of the captured light field.In comparison, because the angular extent of the spatio-temporal lightfield cameras of this invention is determined by the combination of theangular extent of the MLA 220 plus the angular temporal articulation ofthe PDA/MLA assembly 230, the angular extent of the spatio-temporallight field cameras of this invention can be increased without reducingthe signal to noise ratio of the captured light field. The third ofthese advantages is that the spatio-temporal pipelining of themultiplicity of light detection directions described earlier enables thelight detection sensitivity and/or response time the spatio-temporallight field cameras of this invention to be made commensurate withincreased image capture frame rate with minimal latency and withoutsacrificing the captured light field signal to noise as in the case ofprior art light field cameras. The fourth of these advantages is that,through appropriate selection of the angular extent of the MLA 220 plusthe angular temporal articulation of the PDA/MLA assembly 230, thespatio-temporal light field cameras of this invention can be made tocapture the light field with a wide field of view that can reach ±45°.This level of wide field photography cannot be achieved by the prior artlight field cameras without the use of large diameter and rather complexfish-eye wide angle lens systems that, beside significantly increasingthe volumetric size of the light field camera, will also adverselyimpact the prior art light field camera optical performance andsignificantly increase its cost. It should be noted that as a result ofits wide angle light field capture capabilities, the spatio-temporallight field cameras of this invention can be used to capture either wideangle 2D views (panoramic) or 3D views of a wide angle light field.

FIG. 9A and FIG. 9B illustrate the operational principles of thespatio-temporal light field cameras of this invention. FIG. 9Aillustrates an exemplary embodiment of one of the pixel groups G_(i)being comprised of a two dimensional array of n of the pixels of the PDAdevice 210 whereby for convenience the size of the pixel group G_(i)along each axis would be selected to be √{square root over (n)}=2^(m).Referring to FIG. 9A, the directional detection addressability that canbe achieved by the pixel group G_(i) would be accomplished through theaddressability of the n pixels comprising the modulation group G_(i)along each of its two axes x and y using m-bit words. FIG. 9Billustrates the mapping of the axis x and y coordinates of the n pixelscomprising the PDA pixel modulation group G_(i) into individualdirections within the three dimensional light field defined by angularextent Θ of the associated MLA 220 micro lens element such as that ofthe exemplary embodiment illustrated in FIG. 8A. As an illustrativeexample, when the dimensions of the individual pixels of the PDA device210 are (2×2) microns and the PDA pixel group G_(i) is comprised ofn=(2³×2³)=(8×8) pixel array and the angular extent of the associated MLA220 micro lens element is Θ=±15°, then each of the PDA two dimensionalpixel groups G_(i) of size (16×16) micron at the PDA device 210 aperturesurface would be able to detect (8)²=64 individually addressable lightdirections (views) spanning the angular extent of Θ=±15°. When thePDA/MLA assembly 230 is articulated as described earlier (see FIG. 2 andFIG. 4A) using the 2-axis gimbals of the embodiments 600 and 700, thedirectional angular extent provided by the lens elements of the PDA/MLAassembly 230 will be temporally extended by the maximum articulationangle ±α_(max) provided by the gimbal. Thus the directional angularextent provided by the spatio-temporal light field cameras of thisinvention would be temporally extend over an angular coverage totaling±(Θ+α_(max)). For example when the angular extent of the MLA 220 lenselement is Θ=±15°, and the maximum articulation angle α_(max)=±30°, thenthe full angular extent that would provided by the spatio-temporal lightfield cameras of this invention would be (Θ+α_(max))=±45°, and the lightdetection directions (views) it would able to temporally capture wouldbe n[(Θ+α_(max))/Θ]²=9×the number of light directions (views) that canbe detected by the PDA/MLA assembly 230 (see FIG. 5); namely, 9(8)²=576light directions (views). Meaning that the number of views that can becaptured by the spatio-temporal directional light field cameras of thisinvention would be (3×3)n, where n is the size, in terms of number ofPDA device pixels, of the pixel groups G_(i) associated with one of theMLA 220 lens elements. Thus, for this example the spatio-temporaldirectional light modulator of this invention would offer an expandeddirectional detection resolution of 9× the directional detectionresolution provided by PDA/MLA assembly 230. In general, the directionalresolution provided by the spatio-temporal light field cameras of thisinvention would be n[(Θ+α_(max))/Θ]² with an angular extent that extendsover an angle of ±(Θ+α_(max)).

In addition to the directional detection capabilities for thespatio-temporal light field cameras of this invention, spatial detectionwould also be possible using an array of (N×M) of the PDA device 210pixel groups G_(i) such as that described in the previous designexample. If, for example, it is required to create a light field cameraof this invention with spatial resolution of N=256 by M=256 thatprovides the 9(8)²=576 directional detection views of the previousexample, the spatio-temporal light field cameras of this invention wouldcomprise an array of (256×256) directional detection groups G_(i) eachcomprising (8×8)=64 pixels and when a PDA device 210 with (2×2) micronpixel size is used, the PDA device 210 will be comprised of(2048×2048)=4.194 mega pixels, thus making the aperture size of thespatio-temporal light field cameras of this invention be approximately4.1×4.1 millimeter. Using the angular extent values of the previousexample, the spatio-temporal light field cameras of this invention cancapture 576 views of the light field within an angular extent of ±45°with a spatial resolution of (256×256)=65,536 pixels. As explainedearlier, it would also be possible to tradeoff the directionalresolution of the spatio-temporal light field cameras of this inventionfor an increased spatial resolution. For example, in the previousexample if the pixel group size is reduced to (4×4) pixels, thespatio-temporal light field camera of the previous example, using thesame PDA device that is comprised of (2048×2048)=1.048 mega pixels,would provide (512)²=262,144 spatial resolution and (3×4)²=144directional resolution.

As illustrated by the previous examples, the spatial and directionaldetection resolutions of the spatio-temporal light field cameras of thisinvention in terms of the number of individually detectable directionswithin a given angular extent would be determined by selecting theresolution and pixel pitch of the PDA device 210, the pitch of the MLA220 lens elements, the angular extent of the MLA 220 lens elements andthe maximum articulation angle of the camera gimbal. It is obvious to aperson skilled in the art that the MLA lens system can be designed toallow either wider or narrower angular extent, the gimbal design can beselected to allow either wider or narrower articulation angle and thenumber of pixels within each pixel group can be selected to be eithersmaller or larger in order to create spatio-temporal light field camerasthat can achieve any desired spatial and directional detectioncapabilities following the teachings provided in the precedingdiscussion.

The principle of operation of the spatio-temporal light field cameras ofthis invention will be described in reference to the illustrations ofFIGS. 9A and 9B. FIG. 9A illustrates the two dimensional addressabilityof each of the modulation group G_(i) using m-bit resolution for thedirectional modulation. As explained earlier, light detected by the(2^(m)×2^(m)) individual pixels comprising the pixel group G_(i) ismapped by its associated MLA 220 elements into 2^(2m) light directionswithin the angular extent ±Θ of the associated MLA micro lens element.Using the (x, y) dimensional coordinates of the individual pixels withineach of the modulation groups G_(i), the directional coordinates (θ,φ)of a detected light beam is given by:

$\begin{matrix}{{\theta (t)} = {{\alpha_{x}(t)} + {{arc}\; {\tan\left\lbrack \frac{\sqrt{x^{2} + y^{2}} \times {\tan (\Theta)}}{0.5 \times \left( {n - 1} \right)} \right\rbrack}}}} & {{Eq}.\mspace{14mu} 1} \\{{\phi (t)} = {{\alpha_{y}(t)} + {\arctan \left\lbrack \frac{y}{x} \right\rbrack}}} & {{Eq}.\mspace{14mu} 2}\end{matrix}$

Where the α_(x)(t) and α_(y)(t) are values of the articulation anglesaround the x-axis and y-axis at the time epoch t; respectively, theangles θ(t) and φ(t) are the values of the directional sphericalcoordinates of the detected light beam at the time epoch t with thepolar axis at θ=0 parallel to the z-axis of the detection surface of themodulation group G_(i) and m=log₂ √{square root over (n)}, is the numberof bits used to express the x and y pixel resolution within themodulation group G_(i). The spatial resolution of the spatio-temporallight field cameras of this invention is defined by the coordinates (X,Y) of each of the individual pixel group G_(i) within the twodimensional array of modulation groups comprising the overall apertureof the spatio-temporal light field camera. In essence, thespatio-temporal light field cameras of this invention would be capableof temporally capturing a light field described by the spatialcoordinates (X, Y) defined by its detection pixel group array and thedirectional coordinates (θ,φ), with the latter being defined by thevalues of the coordinates (x, y) of the pixels within the detectiongroup G_(i) and the temporal value of the articulation angle of thespatio-temporal light field cameras as defined by Eq. 1 and 2 above.

Another distinctive aspect of the spatio-temporal light field cameras ofthis invention is that unlike prior art light field cameras describedearlier which typically capture a set of planar stationary parallaxes ofthe light field, the spatio-temporal light field cameras of thisinvention capture curved temporal parallaxes of the light field. FIG. 9Cillustrates the curved temporal parallax that can be captured by thespatio-temporal light field of this invention. As illustrated in FIG.9C, because of its temporal angular articulation, the parallaxes of thelight field captured by the spatio-temporal light field cameras of thisinvention are actually curved both temporally and in each of the2-dimensional spatio-directional (X, θ) or (Y, φ) parallaxes (slice) ofthe light field it captures. Because of this distinctive feature, thespatio-temporal light field cameras of this invention can capture acurved wavefront, which is a more natural way of recording light fieldinformation than prior art light field cameras as most optical devices,including the human visual system (HVS), have a naturally curved imageplane. Furthermore, the captured information includes temporal samplesof the light field, which means that the spatio-temporal light fieldcameras of this invention can record information not only about thespatial and directional aspects of the light field but also about thephase of the wavefront of the light field it captures. In essence,therefore, the spatio-temporal light field cameras of this invention canrecord curved parallaxes of the 4-dimensional spatio-directional spaceof the light field plus the phase of its wavefront. In other words, thespatio-temporal light field cameras of this invention can record all5-dimensions of the light field; two spatial dimensions, two directionaldimensions plus phase. Numerous light field photography and displaycapabilities will be enabled by this 5-dimensional light field featureof the spatio-temporal light field cameras of this invention, a few ofwhich are discussed in subsequent paragraphs of this disclosure.

Any desired spatial and directional detection capabilities can berealized using the spatio-temporal light field cameras of thisinvention. The previous examples illustrated how spatio-temporal lightfield cameras of this invention with (256)² spatial resolution and(3×8)² directional resolution can be implemented using a single 4.1×4.1millimeter PDA device 210. One possible approach to realize thespatio-temporal light field of this invention with a higher spatialresolution can be achieved using a higher pixel resolution PDA device210. If for example a PDA device 210 that comprises a (512×512) of then=64 pixel groups is used, meaning a PDA device that is comprised of(4096×4096)=16.8 mega pixels, the spatio-temporal light field camera ofthe previous example of this invention can capture 576 views of thelight field within an angular extent of ±45° with a spatial resolutionof 262,144 pixels. In this case the full aperture of the spatio-temporallight field cameras of this invention would only be approximately8.2×8.2 millimeter. Another possible approach to realize even higherspatial resolution spatio-temporal light field cameras of this inventioncan be achieved by tiling a multiplicity of smaller spatial resolutionPDA/MLA assemblies 230 of this invention. For example, when an array of(3×3) of the PDA/MLA assemblies 230 of the previous example are tiled asillustrated in FIG. 10, the resultant spatio-temporal light fieldcameras would provide (3×512)² spatial resolution (more than 2.35 megapixels) with (3×8)²=576 directional resolution. The full aperture sizeof the spatio-temporal light field cameras of this invention in thiscase would be approximately 2.46×2.46 centimeter, but its thicknesswould still be approximately 5 millimeter. The tiling of a multiplicityof the spatio-temporal light field cameras of this invention in order torealize a higher spatial resolution version is possible because of itscompact volumetric dimensions. For example, the spatio-temporal lightfield camera of the previous example that uses a single PDA device 210,which by itself would have a width, height and thickness of 8.2×8.2×5mm; respectively, can be used to create the larger resolution versionillustrated in FIG. 10 which would have the dimension of 2.46×2.46×0.5cm in width, height and thickness; respectively. It would be possible toimplement the higher spatial resolution version of the spatio-temporallight field cameras of this invention illustrated in FIG. 10 by bondinga multiplicity of the PDA/MLA assemblies 230 of the previous example toa backplane using electrical contacts of the micro ball grid array(MBGA) located on its backside, which given the zero-edge feature of theembodiment 700 of this invention, would make it possible to realizeseamless tiling of a multiplicity of such light field capture devices toimplement any desired size of the spatio-temporal light field cameras ofthis invention. Of course the size of the array of PDA/MLA assemblies230 illustrated in FIG. 10 can be increased to the extent needed torealize any desired spatial resolution. It is worth noting that thetiling of PDA/MLA assemblies 230 in the spatio-temporal light fieldcameras of this invention in order to realize the expanded spatialaperture illustrated in FIG. 10 is made possible by the zero-edgefeature described earlier of the embodiment 700 of this invention.

FIG. 11 illustrates an exemplary embodiment of the data processing blockdiagram of the spatio-temporal light field cameras of this invention. Asillustrated in FIG. 11, the output data from the spatio-temporal lightfield cameras of this invention will be formatted in multiple bit wordswhereby each output word contains three data fields. The first datafield is the address (X, Y) of spatial group G_(i) within the detectionpixel group array comprising the aperture of the spatio-temporal lightfield cameras of this invention. This first data field will in effectrepresent the spatial address of the output of the spatio-temporal lightfield cameras of this invention. The two remaining data fields providethe data representation of the light detected by the spatio-temporallight field cameras of this invention in each of the directionalcoordinates (θ,φ) for each of the spatial coordinates (X, Y). These twodata fields will in effect represent the directional output of each ofthe spatial coordinates (X, Y) of the spatio-temporal light fieldcameras of this invention. Referring to FIG. 11, the data processingblock 120 processes the output of the PDA device 210 and decodes thepixel output address data to derive the spatial addresses (X, Y) and thedirectional addresses (x, y). In the data processing block 130 of FIG.11, the directional addresses (x, y) are then augmented with twoadditional data fields that represent the instantaneous values of thearticulation angles α_(x)(t) and α_(y)(t). In the data processing block140 of FIG. 11, the directional values (x, y) are combined with theinstantaneous values of the articulation angles α_(x)(t) and α_(y)(t)using Eq. (1) and (2) to generate the directional coordinate values(θ,φ). The data processing block 150 concatenates the detected lightintensity and color data fields of data output values of the pixels ofthe PDA device 210 with the mapped spatial addresses (X, Y) anddirectional coordinates (θ,φ) to generate the three data fields: F1=thespatial addresses (X, Y), F2=the directional coordinates (θ,φ); andF3=the detected light intensity and color data.

In using a 16-bit word for representing the directional coordinates(θ,φ) of the detected light field and the typical 24-bits forrepresenting the modulated light intensity and color in each direction,the total number of bits that would represent the detected light fieldfor each spatial address (X, Y) would be 40-bit words. In assuming,without loss of generality, such 40-bit words would be outputted fromthe spatio-temporal light field cameras of this invention sequentially;i.e., sequential addressing is used to output the 40-bit words, block120 of FIG. 11 would be responsible for routing the sequentiallyoutputted data word from the designated PDA device. Block 150 of FIG. 11would be responsible for formatting and outputting the 40-bit words ofthe F2 and F3 data fields of the detected light field data sequentiallyfor each of the spatial coordinates (X, Y). It should be noted that theentire data processing flow illustrated in FIG. 11 will be executed onceper capture frame period. In using the previous example in which thespatial resolution of the spatio-temporal light field cameras of thisinvention is (256×256), and in assuming the frame capture period is16.67 millisecond (which is equivalent to 60 Hz frame rate), thespatio-temporal light field cameras of this invention would output256×256×40˜2.62 megabit per the 16.67 millisecond frame cycle; which isequivalent to approximately 157 Mbps output data rate. With thisexemplary data processing flow of the 40-bit word sequential dataoutput, the spatio-temporal light field cameras of this invention woulddetect the light field that enters its aperture in intensity, color anddirection and output the detected light field the information encodedwithin its output data.

Possible Applications

3D Camera—

The spatio-temporal light field cameras of this invention can be used toimplement a 3D camera with arbitrary spatial and directional resolutionsthat is realized, for example, as a tiled array of a multiplicity ofPDA/MLA assemblies 230 such as that illustrated in FIG. 10. Such a tiledarray functionally is the same as a much larger PDA/MLA assembly. Theexpanded full angular extent that can be realized by the spatio-temporallight field cameras of this invention would enable the realization of 3Dcameras that are volumetrically compact and capture a large field ofview, yet without the use of bulky and costly optical assemblies as withthe case of prior art light field cameras. The level of volumetriccompactness that can be achieved by the spatio-temporal light fieldcameras of this invention will enable the realization of ultra compact3D light field cameras that can be embedded in mobile devices such cellphones and tablet PC and the like. In addition the 3D light fieldcaptured by the spatio-temporal light field cameras of this invention asrepresented by the output format described in the preceding discussion,would be directly compatible with the class of light field modulatorsdescribed in U.S. patent application Ser. No. 13/329,107 entitled“Spatio-Optical Directional Light Modulator” and Ser. No. 13/546,858entitled “Spatio-Temporal Directional Light Modulator”, both assigned tothe Assignee of the present application, thus making it possible tocapture the 3D light field using the spatio-optical light field camerasof this invention, then directly displaying it using the light fieldmodulators described in the applications referenced above.

Computationally Focused 2D Camera—

The spatio-temporal light field cameras of this invention can also beused to capture 2D light field images, either still or video, that canbe computationally (or digitally) focused. In this case the output ofthe spatio-temporal light field cameras of this invention described inFIG. 11 will be processed by an image processor to create an image thatis focused on any plane or surface within the captured light field. FIG.12 is an illustration of the light field that would typically becaptured by the spatio-temporal light field cameras of this invention ina two dimensional slice (X, θ) across its spatial dimensions (X, Y) andits directional dimensions (θ, φ). The trapezoids 181, 182 and 183represent the light field captured in the parallax slice (X, θ) fromthree objects at different distances from the spatio-temporal lightfield cameras of this invention. The trapezoids 181 represents the lightfield from the object nearest to the light field camera and thetrapezoids 183 represents the light field from the object farthest fromthe camera while trapezoids 182 represents the light field from anobject at an in between distance. As can be seen from FIG. 12, thedistance of the object from the spatio-temporal light field cameras ofthis invention, and hence the depth of the field, is encoded by thecamera as the inclination of the light field captured by the camera ineach parallax slice within the captured light field. Accordingly thedepth information captured by the spatio-temporal light field cameras ofthis invention can be utilized to computationally create a 2D image thatis focused on any desired object, plane, curved surface or even a 3Dobject within the captured light field. Such a computational (digital)focusing can be accomplished using established computational photographyprinciples in which the sub-images of the light field captured by theindividual lens elements of the MLA 220 are first computationally scaledby a factor that is proportional to the distance of the desired focusplane from the aperture of the camera, then the scaled sub-images areshifted in their (x, y) coordinates by a factor proportional to thepitch distance between the lens elements of the MLA 220, then addedtogether to create an image that is focused at the desired plane orcurved surface. It would also be possible to computationally create fromthe light field captured by the spatio-temporal light field cameras ofthis invention an image with any desired depth of focus. In this case amultiplicity of images focused on multiple planes or curved surfaceswithin the desired depth of focus are computationally created, asexplained earlier, then summed to create a unified image which wouldthen have in focus all of the imaged objects within the computationallycreated depth of focus (i.e. the depth volume defined by the depth offocus and the multiple planes of curved surfaces). It should be notedthat it would also possible to computationally create from the lightfield captured by the spatio-temporal light field cameras of thisinvention an image that is focused on a curved surface. This capabilityis made possible because, as explained earlier (see FIG. 9C), thespatio-temporal light field cameras of this invention inherently capturecurved parallaxes of the light field through the temporal angulararticulation of their apertures, which creates a curved wide angle fieldof view. With this curved light field capture capability it would bepossible to computationally create from the light field captured by thespatio-temporal light field cameras of this invention an image that isfocused on multiplicity of objects at different distances from thecamera yet with rather narrow depth of focus, which would consequentlyenables the capability of computationally creating higher resolutionimages of multiplicity of objects at different distances from the cameraaperture.

Switchable 2D/3D Camera—

It is also possible for the spatio-temporal light field cameras of thisinvention to be switched from 2D to 3D display modes by adapting theformat of its output data described earlier (see FIG. 11) to becommensurate with the desired operational mode. In either operationalmodes captured light field angular extent will be that associated withits MLA 220 micro lens element plus the articulation angle of its gimbal±(Θ+α_(max)) with the pixel resolution of the individual modulationgroup G_(i) and the maximum articulation angle α_(max) definingdirectional resolution of the camera, as described earlier, modulationgroups G_(i) defining its spatial resolution.

Networked Light Field Photography—

As stated earlier, the volumetric advantages of the spatio-temporallight field cameras of this invention make it possible to embed such acamera in mobile devices such cell phones and tablet PC. Since most allof such host mobile devices are typically interconnected either throughwireless or wireline networks or through bulk data transfer using flashmemory modules, it is possible to leverage such connectivity to furtherextend the light field captured by the light field cameras of thisinvention. In this embodiment, the output of a multiplicity of embeddedspatio-temporal light field cameras of this invention located in thesurroundings of a viewing scene that captured light field images of theviewing scene, as illustrated in FIG. 13, will have the datarepresenting the light field it captured (described in FIG. 11). Thatdata can be augmented with three additional data fields by applicationsoftware incorporated into the operating system of the host mobiledevice. The first data field would specify the camera location, whichwould typically be the output of the location sensing device alsoembedded in the host mobile device, such as a global positioning system(GPS) receiver or triangulated wireless link; the second data fieldwould specify the orientation of the mobile device, and thus theorientation of its embedded camera, which would typically be the outputof an orientation sensing device also embedded in the mobile device,such as the micro gyros typically embedded in mobile devices foradapting the display screen orientation and for gamming; and the thirddata field would specify the time the light field of the viewing sceneis captured by the camera, which would typically be the output of theinternal mobile device clock which is typically kept synchronized with anetwork time. When this augmented light field data is exchanged withother mobile devices also having an embedded spatio-temporal light fieldcamera of this invention that has captured the light field of the sceneviewed from a different perspective, it would be possible to integratethe exchanged light field data captured by the multiplicity of suchcameras and computationally fuse it together into a single super lightfield data set that represent the collective light field captured by allthe embedded cameras that captured a partial perspective of the viewingscene. The computational aspects of this “networked light fieldphotography” would involve making use of the location, orientation andtime of capture data fields augmenting each exchanged or networked lightfield data to transfer such exchanged light field data from thecoordinate of the set of respective embedded cameras that captured it toa set of viewing scene coordinates that would be used as the commoncoordinates of the networked light field. Once the light field datatransformation (or fusion) is performed, the transformed collectivelight field data is exchanged back to all of the participating mobiledevices for sharing the entire light field as captured collectively byall of the participating host mobile devices. The operational concept ofthe described networked light field photography embodiment isillustrated in FIG. 13.

Although reference is made in the preceding paragraph to a networkedmobile device; such as cell phones and tablet PCs for example, being ahost of the spatio-temporal light field cameras of this invention toenable the networked light field photography embodiment illustrated inFIG. 13, a person skilled in the art would know that it would be obviousto implement the networked light field photography embodiment of thisinvention by creating a networked light field camera device having itsown integrated networking capabilities rather than having the lightfield camera being hosted or embedded into a networked mobile devicesince either approaches would be functionally equivalent.

The light field data exchange and the computational aspects of thenetworked light field photography described in the previous paragraphcan be performed in one of multiple ways: (1) the superset of exchangedaugmented light field data could be processed by each individualparticipating mobile device to generate the collective light field thataggregate the light fields of the viewing scene captured by all of theparticipating networked mobile devices; (2) the superset of exchangedaugmented light field data could be partially processed by eachindividual participating mobile device in a processing load sharingstyle until the transformation of the exchanged data converges, possiblyafter multiple intermediate data exchanges through data networking, intothe aggregate collective light field of the viewing scene captured byall of the participating networked mobile devices collectively; or (3)the superset of exchanged augmented light field data could be processedby a networked server that receives the augmented light field data sentthrough the network from all of the participating mobile devices withthe embedded cameras that captured the viewing scene, then the servertransforming the received data to generate the collective light fieldthat aggregates the light field of the viewing scene captured by all ofthe participating networked mobile devices, with the server thendownloading the transformed collective light field data back to theparticipating mobile devices. It should be noted that data compression,in particular compression schemes that takes advantage of the spatial,direction and temporal correlation inherently present within capturedlight field data, would be used throughout the exchange of networkedlight field photography process.

It is worth mentioning that the embodiment of the networked light fieldphotography described in the preceding paragraphs would also beeffective, albeit at somewhat lower light field capture resolution, evenwhen the output of conventional digital cameras embedded in mobiledevices is augmented with the location, orientation and time of capture,and is used to create a collective light field of a scene captured by amultiplicity of participating mobile devices networked and processed asdescribed earlier. The use of the spatio-temporal light field cameras ofthis invention will further enhance the collective light field captureresolution that results from the networked light field photographydescribed in the preceding paragraphs. It will also be possible to makeuse of combinations of augmented data from images captured byconventional digital cameras embedded in networked mobile devicestogether with the augmented light field data captured by spatio-temporallight field cameras of this invention embedded in networked mobiledevices within the context of the networked light field photographydescribed earlier. In this case both types of cameras would contributeimages of the viewed scene captured from different perspective while theaugmented data contributed by spatio-temporal light field cameras ofthis invention would in addition represent the entire light field in itsfield of view. In general, the networked light field photographyembodiment described in the preceding paragraphs, using either thespatio-temporal light field cameras of this invention, conventionaldigital cameras embedded in mobile devices or a combination of bothtypes of camera, would be effective in 3D capture of social events (suchas sports games, concerts, etc. . . . ), and can also be effective in 3Dsurveillance and 3D cinema capture.

When the computational processing of the captured and augmented lightfield data is performed by a networked server as described in thepreceding paragraphs, the networked light field photography can beapplied within the context of an internet application that would allowits subscribers to participate by uploading to the server the lightfield data captured using their mobile device with an embeddedspatio-temporal light field camera of this invention, then be able todownload the collective light field computationally fused by the serverfrom the light field data captured by all of the subscribers attendingthe same social event. In essence with the networked light fieldphotography, the participants who each captures the scene from adifferent perspective (where they stand relative to the viewing scene)using their embedded spatio-temporal light field camera of thisinvention would collaborate through their server connectivity tocollectively create a light field of the viewing scene that allparticipants can share by downloading the fused collective light fieldfrom the server.

In the alternative approach described earlier in which the computationalprocessing of the augmented light field data is performed by eachindividual participants' mobile device or other personal computationalassets like a PC for example, the fused collective light field data ofthe viewing scene can still be shared among the participants usingeither the same media through which their captured light field data wasoriginally shared or through internet accessible sites such as a blog ora social network site. In essence with this alternative approach of thedescribed networked light field photography embodiment the participantswho each captures the viewing scene from a different perspective (wherethey stand relative to the viewing scene), using their embeddedspatio-temporal light field camera of this invention, would exchangetheir captured and augmented light field data using either a mobilewireless telephone network, wireless local network (WLAN), such as WiFifor example, a personal area network (WPAN), such as Bluetooth forexample, or through bulk data transfer using flash memory modules, thenuse their personal computing assets in order to fuse the shared lightfield data to create the collective light field data which they in turncan share with other participants through one of the media used toexchange the individually captured and augmented light field data. Itshould be mentioned that the latter approach of the networked lightfield photography can also be applied to create 2D wide angle views (orpanoramas) of a viewing scene which would be captured collaboratively bymultiple participants using their networked mobile devices with anembedded spatio-temporal light field camera of this invention.

The above camera embodiments, networked or not, record a fourdimensional light field (X and Y positions and angles to the sources ofthe light) that when displayed using the light field data from that onecamera or using the fused collective light field data of the viewingscene from the described networking, create a three dimensional image ofthe scene viewed by the camera. Such a three dimensional image may beviewed from any angle and still present a three dimensional image. Byway of example, a horizontal image might be displayed at a table top orcoffee table level. Such an image will be a three dimensional imagewithin the field of view of the camera that took the image when viewedby persons standing anywhere around the display.

As an alternative, some three dimensional images might be viewed in apredetermined manner rather than from any angle around the image. By wayof example, a three dimensional image might be displayed in a verticalplane, and viewed sitting down or standing up by persons lookingstraight at the image without significant tilt of their heads. In thatapplication of images recorded by the camera, the three dimensionalappearance of the images in the vertical direction is not perceived bythe viewers, so that the recorded images need only have recorded thehorizontal angles to the sources of the light over the field of view ofthe camera. The recording of the total light from any point in theviewing scene in the vertical direction is all that is essential for thethree dimensional effect, as the angles to the sources of the light inthe vertical direction are not perceivable when a viewers eyes are kepthorizontal. Such an embodiment can simplify the camera.

As an example, if the photo detector array has a sufficient number ofpixels to provide the desired resolution in the vertical direction, thenthe photo detector array need not be articulated about two axes, butinstead articulated about one axis only, in this example, the verticalaxis so as to only expand the horizontal field of view and/or thespatial resolution. Instead of using a 3×3 tiled array of PDA/MLAassemblies 230, one could use a 3×1 array, so that the articulationcapability, articulation actuators, articulation sensors andinterconnects for one axis are eliminated.

Of course in such applications, as a further alternative, the full threedimensional image may be recorded, and then before or on playback, theimage data may be processed so that the three dimensional effect in thevertical direction is eliminated, in which case the light at any pointin the image would be displayed without consideration of the angle inthe vertical plane from which the light originated within the field ofview of the camera.

In summary, the foregoing description presented multiple embodiments ofnovel light field cameras that overcome the limitations and weaknessesof prior art light field cameras and make it feasible to create lightfield cameras that can be used to record either still or motion videoimages, that can be embedded in mobile devices, and offer the users ofsuch devices the capability of computational focusing of 2D images andthe capture of 3D images both over a wide angular extent, and to providethe capability for networked light field photography. Of course,features and implementations of any embodiment may be incorporated inany other embodiment, and are not limited to use with respect to anysingle embodiment.

Thus the present invention has a number of aspects, which aspects may bepracticed alone or in various combinations or sub-combinations, asdesired. While certain preferred embodiments of the present inventionhave been disclosed and described herein for purposes of illustrationand not for purposes of limitation, it will be understood by thoseskilled in the art that various changes in form and detail may be madetherein without departing from the spirit and scope of the invention asdefined by the full breadth of the following claims.

What is claimed is:
 1. A method comprising: providing a plurality oflight field cameras each having; a micro photo-detector array devicehaving a light detection surface and a micro lens array of micro lenselements; whereby the micro photo-detector array device and the microlens array are assembled together and temporally angularly articulatedas a single assembly around two orthogonal axes parallel to a plane ofthe light detection surface of the micro photo-detector array device;the temporal angular articulation being within a range of plus to minusa maximum angular articulation for each axis; embedding each light fieldcamera in a respective mobile device; interconnecting each mobile deviceto at least one host device through wireless or wireline networks orthrough bulk data transfer using flash memory modules; exchanging lightfield camera output data between mobile devices that captured adifferent perspective of a viewing scene and the host device to formcollective light field camera output data; and using the collectivelight field camera output data to create a collective light field of theviewed scene.
 2. The method of claim 1 wherein each light field camerais provided with a tiled array of the micro photo-detector array devicesand micro lens arrays.
 3. The method of claim 1 wherein the mobiledevices are cell phones, tablet PCs or light field cameras with theirown connectivity capabilities.
 4. The method of claim 1 wherein formingthe collective light field camera output data is performed by either ofthe following ways: (1) processing the collective light field cameraoutput data by each mobile device to generate the collective light fieldwherein each mobile device acts as a host device; (2) wherein the hostdevice is a network server, and processing the collective light fieldcamera output data by the network server, then downloading thecollective light field of the viewed scene to the mobile devices.
 5. Themethod of claim 1 wherein for each light field camera in each mobiledevice: the micro photo-detector array device comprises an array ofpixels whereby each of the pixels is a few microns in size and isindividually addressable through the device interface contacts to outputan electrical signal that is responsive to color and intensity of lightcoupled into its aperture, the angular articulation temporally expandsthe angular extent by twice the maximum angular articulation and therebytemporally expands a detectable light field size and increases thenumber of light detection directions by the ratio of said angular extentexpansion to the angular extent along each of the two articulation axes,the individual pixel color and intensity detection addressability andthe temporal association of the pixels within a pixel group with thetemporally expanded set of light directions enabling the detection ofthe color and intensity of the light impinging from each of thetemporally expanded set of directions, thereby enabling the light fieldcamera to detect the color, intensity and direction of the light thatimpinges its aperture from the expanded light field defined by thediscrete set of directions comprising the set of the temporally expandedset of light directions, and the directional, spatial, color andintensity detection addressability of the light field camera using amultiple field data output from the light field camera for light fieldcamera output data whereby for each designated pixel group addresswithin the spatial array of the pixel groups, at least one output datafield is used to specify the direction of the light impinging theaperture of the light field camera and at least one output data field isused to specify the color and intensity of the light detected from thatdesignated direction; and wherein the exchange of the light field outputdata of the plurality of light field cameras and the host device tocreate the collective light field camera output data is accomplished by:first, augmenting the light field output data of each light field camerawith data representing the camera location and orientation and capturetime of the light field to form augmented light field output data,second, exchanging, through the interconnectivity of the host device,the augmented light field camera output data to form a superset ofaugmented light field camera output data, third, computationally fusingthe superset of augmented light field camera output data to create acollective light field of the viewed scene, and forth, exchanging,through the interconnectivity of the host mobile device the collectivelight field of the viewed scene with the plurality of mobile devices. 6.The method of claim 5 as applied to create two dimensional wide angleviews or panoramas of a viewing scene captured collaboratively bymultiple participants using their interconnected mobile devices, eachwith an embedded light field camera.
 7. The method of claim 5 whereincomputationally fusing the superset of augmented light field cameraoutput data to create a collective light field of the viewed scenecomprises converting the light field data of each of the mobile devicesfrom the coordinates of the respective light field cameras to a set ofviewing scene coordinates.
 8. The method of claim 5 whereincomputationally fusing the superset of augmented light field cameraoutput data to create a collective light field of the viewed sceneincludes computationally fusing two dimensional digital camera outputdata to create a collective light field of the viewed scene.
 9. Themethod of claim 8 wherein the computational light field data fusing isperformed by a networked server using an internet application thatallows subscribers to upload to the server the augmented light fielddata captured by the light field camera or corresponding data capturedby a two dimensional digital camera embedded in the respective mobiledevice, then to download the collective light field computationallyfused by the server from the data captured by all of the subscribersattending the same event.
 10. The method of claim 5 used to capturethree dimensional light field images of social events, for threedimensional surveillance and three dimensional cinema capture.
 11. Themethod of claim 5 wherein the computational light field data fusing isperformed by each individual participant's mobile device or otherpersonal computational assets of the individual participants to generatethe collective light field of the viewing scene which in turn is sharedamong the participants using either the media through which theircaptured light field data was originally exchanged or a different media.12. The method of claim 11 wherein the captured and augmented lightfield data and the fused collective light field data are exchanged amongthe participants of the networked light field photography using one ormore of a mobile wireless telephone network, wireless local network, apersonal area network, and bulk data transfer using flash memory modulesor an internet accessible site.